For too many similar words and lots of irregular writing ways of the same words in the handwritten character recognition, a modified Affinity Propagation (AP) clustering algorithm was proposed to add to the recognition process. Clustering judging function Silhouette was combined with original AP algorithm in the proposed algorithm. Class number was updated by changing preference parameter adaptively through iterative process of AP algorithm. And then the optimal clustering result was obtained by assessing clustering quality of every iteration. The experiment of handwritten Chinese character recognition indicates that the recognition rate of recognition process added original AP algorithm is 1.52% higher than the rate of traditional recognition process. And the recognition rate of recognition process added modified AP algorithm is 1.28% higher than the rate of recognition process added original AP algorithm. The experimental results verify that it is effective to add clustering algorithm to the handwritten character recognition process. And compared with original AP algorithm, convergence and clustering quality of modified AP algorithm are also improved.